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ABSTRACT 

Research findings are reported that suggest that 
valid interpretation of assessment scores on illiterate and 
preliterate learners requires the use of ethnographic data. Data from 
observation notes, photos, and audiotapes indicated that learners' 
understanding of their tasks affected their performance in assessment 
situations. Previous findings from studies involving 
Engl ish-as-a~Second-Language (ESL) learning suggests the need to 
combine qualitative and quantitative research methods. The following 
recommendations are offered for procedures in cross-cultural 
educational research: (1) supplement written information about the 
subjects 1 culture with interviews with cultural informants and 
participant observations; (2) conduct a pilot study with individuals 
from the target culture; (3) determine how subjects view their 
participation in the study and anticipate a high number of 
noncompleters ; (4) check the reliability of translators* messages to 
subjects; (5) accommodate cultural patterns in procedures; (6) verify 
subjects 1 familiarity with media used in the study; and (7) 
contextual ize quantifiable outcomes with the ethnographic data 
collected during the study. Future directions for research on ESL 
literacy are noted. Appended materials include tables displaying 
quantitative data, a sample of translated documents (post-instruction 
interview), sample observation notes, and a sample drawing. Contains 
142 references. (Adjunct ERIC Clearinghouse on Literacy Education) 
(LB) 
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BESHITrOSLAE: 



INTERPRETING ASSESSMENT SCORES OF NONLTTERATE LEARNERS WITH 

ETHNOGRAPHIC DATA 



Abstract 



This paper presents research findings which indicate that valid 
interpretation of assessment scores on illiterate and preliterate learners requires the 
use of ethnographic data. Data from observation notes, photos, and audio tapes 
indicate that learners' understanding of their tasks affected their performance in 
assessment situations. 

Introduction 

Each year, hundreds of research studies are published which use English as 
a second language students as subjects. Data reported on the subjects' 
performances ultimately inform evolving theories that guide second language 
instruction. 

Because of the potential impact of our studies on pedagogy, researchers 
must be certain that the data we collect are valid representations of subjects 1 
performances. Some questions that might guide us in this endeavor are the 
following: 

1. How familiar are our subjects with schooling traditions inherent in our research 
procedures? 

2. Do the research methods we are using fairly assess our subjects' performances 
on second language tasks? 

3. What steps have we taken to ensure that subjects in our research studies 
understand their tasks? 



4. How do know whether the materials used in our studies impede or facilitate 
subjects 1 performances? 

5. What do we know about the culture(s) of our subjects which may affect their 
performance as learners? 

6. How do we counterbalance our cultural biases when we observe and interpret 
our subjects 1 behaviors? 

7. From whom do we seek guidance to ensure that our procedures are sensitive 
and appropriate to the cultures of our subjects? 

8. How cautious are we in interpreting results of our studies? 

9. How much do we sensitize our colleagues to the influence of culture on 
subjects' performances in learning situations? 
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Information about subjects' culture(s) 

These questions suggest that our preparation for ESL research projects must 
include some knowledge of the schema which our subjects bring to the task at 
hand. Such information necessarily includes general knowledge of the culture(s) of 
the students who will serve as subjects, as well as analysis of ways in which 
procedures we plan to use are compatible with or in opposition to subjects 1 
dominant cultural patterns. The most common cultural conflict is between the 
requirement for individual performances without the opportunity to consult other 
learners from subjects' whose culture(s) value collaboration and consultation. If 
we are not familiar with the culture(s) of the subjects in our research, we need to 
seek the assistance of cultural informants and/ or read about cultural behavior 
patterns. 

When a multi-cultural ESL class constitutes our subject pool, this 
preparation activity becomes time-consuming and complex. However, findings 
from cross cultural studies such as those by Pettersen (1984), Cibrowski (1980), 
Triandis and Berry, Jahoda (1980), Pick and Pick (1978), and indicate that we can 
make serious errors in interpreting data from subjects from different cultures if we 
do not have this information. 

How many times have we reported data from an ESL class by their 
proficiency level without noting the variety of cultures represented within that 
class? Findings by Hvitfeldt ( 1985) and Holloway (1990) regarding Hmong and 
Mien students' misinterpretation of two dimensional drawings (see Appendix A-4) 
indicate that we need to be particularly cautious in interpreting studies which use 
visual media. 

The presence of a reliable, bilingual cultural informant throughout the study 
ensures that materials and tasks in the research project are understood by the 
subjects, and that we, as principal investigators, are alerted to materials and /or 
procedures which are causing stress or conflicting with subjects' cultures in ways 
serious enough to affect the validity of the data. 

Ethnographic data 

One way of ensuring that our research data are accurate is to get a complete 
description of the context of the research situation. This includes collecting 
observation notes, making audio and video tapes, taking photographs, and 
conducting interviews with the subjects in their own language. Such data allow 
the researcher to analyze subjects' atientiveness to and comfort level with materials 
and procedures in learning tasks. These data are valuable in helping explain why 
subjects performed well or poorly. 



4 



S. Griffin, 4/7/93, 2 



Combining qualitative and quantitative research methods 



In my study, The Effect of Media Choice on English Literacy Acquisition 
for Preliterate Adult Hmong Learners, I combined qualitative and quantitative 
methods, to examine media choice and the effects of three kinds of media (video, 
photographs, and live teacher demonstrations with real objects) on English literacy 
acquisition for 37 Hmong adults enrolled in ESL classes. Examples of the 
observational data are included in the appendix of this study. 

Finding s 

The quantitative data from the study (see Appendix A-l), when interpreted 
separately from the qualitative date, were not impressive. Analysis of Covariance 
(ANCOVA) showed that subjects 1 pre-instruction scores on the literacy section of 
the Washington State Adult Refugee Project ESL Oral Placement Test strongly 
related to their performance on the post-instruction literacy achievement test. 
When pre-instruction literacy was used as a control variable in subsequent 
analyses of the data, choice of instruction media (subject's own selection of 
instructional media, rather than assignment to media by the teacher) had no 
significant effect on English literacy acquisition. Other variables which did not 
significantly affect English literacy acquisition were: the match between subjects* 
preferred perceptual modes and the characteristics of instructional media which 
appeared to appeal to these modes, amount of previous formal ESL instruction, 
and the subject's age and sex. 

However, observational data, in combination with subjects' achievement test 
mean scores, yielded surprising results. For subjects who were familiar with these 
media, bilingual video and colored photos (in combination with flash cards) were 
more effective than teacher demonstrations in promoting English literacy 
acquisition. These findings showed that the prevailing use of live teacher 
demonstrations in the instruction of nonliterate second language literacy learners 
(Green and Reder, 1986; Weinstein, 1984, 1986; Reder, 1982, 1984; and Robson, 
1982) may successfully promote their speaking and listening skills, but actually 
inhibit their acquisition of literacy skills. 

Observational data (see Appendix A-3) and photos showed that subjects 
were distracted from attention to the flashcards and the task of linking written 
language on the flashcards with the sounds of the words and meanings being 
conveyed by the objects and actions being demonstrated. Instead, they focused on 
the aspects of the communication situation which would predictably engage 
individuals from an oral tradition culture: nonverbal signals from the teacher, the 
teacher's clothing, and the social setting of the classroom. Subjects appeared 
overloaded with verbal and nonverbal information, making attentiveness to 
literacy nearly impossible. 

The non-pictorial, bilingual video tape, reduced personal distractions and 
focused learners on the connection between speech and writing. The use of 
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translation supplied the learners with instant understanding of the meaning of 
each sentence. The photographs, although not appealing to learners because of 
the use of disembodied hands holding objects, also focused learners 1 attention on 
the link between the written word, the sound of the words and meaning. 

Other strong influences on subjects 1 English literacy acquisition in this study 
were their familiarity with technologies and symbolic codes of media used in 
instruction (photographs, television, and flash cards), and their understanding of 
how their instructional activities related to acquisition of English literacy. Analysis 
of audio tapes indicated that, despite repeated explanations of the study through 
Hmong translators, subjects regarded this experience as a new kind of ESL class. 
They repeatedly questioned how these activities related to learning English. Not 
only did subjects not have a good understanding of the culture of school, they had 
no understanding (How could they?) of the concept of an educational experiment. 

Implication s 

Without the observational data from this study, we might never have 
known why the teacher demonstrations were not successful in promoting literacy. 
We might have speculated about the rapport between the teacher and the subjects 
or have focused our attention on subjects' lack of familiarity with some of the 
objects used in the teacher demonstrations. However, the data did provide us with 
valuable information about the effect of such demonstrations on attentiveness to 
the target information in the lesson: English in written form. 

Other data collected provided valuable information about the reaction of 
this group of Hmong subjects to standard procedures used in formal research. 
They found formal test booklets strange and puzzling. They felt a need to 
discuss the instructions given by the interpreter before proceeding with any task. 
They became anxious during one-to-one interviews conducted in private, but were 
quite comfortable being interviewed in the midst of a noisy background with 
many distractions. Finally, they were bemused at the idea of being asked about 
their preferences for different learning experiences in light of their recent years of 
taking orders from others in refugee camps and welfare offices. 

Recommendations for research colleagues 

Contributions to Cross-Cultural Methods of Educational Research 

My study demonstrated the importance of addressing the cultural and conceptual 
gaps between the assumptions and procedures of modern, Western educational 
research and subjects from non-Western, non-technological cultures in the design 
and procedures of the study, in the kinds of data collected, and in the analysis of 
the data. 

As already indicated, a principal lesson learned from this study is that test scores 
cannot give a complete picture of what subjects from a different culture know 
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about the subject matter being tested. When conducting studies using subjects 
such as the Hmong, who have no formal education and no context for 
understanding educational experiments, the researcher must build into the 
research design several ways to determine what subjects understand about their 
tasks. 

Recommendations for Procedures in Cross-Cultural Educational Research 

The data from this study suggest some specific recommendations for 
educational researchers intending to work with populations from cultures who are 
not familiar with Western educational practices. A number of these 
recommendations apply to classroom-based research some of the general 
guidelines found in Handbook of Cross-Cultural Psychology edited by Triandis 
and Berry (1980): 

1. Supplement written information about the subjects' culture with interviews of 
cultural informants and participant observations. Accurate cultural information is 
essential in planning activities of cross-cultural studies and in interpreting the data. 
The researcher's contacts with the Hmong community in Seattle, observations of 
preliterate Hmong adults in classroom and social settings, and discussions with 
cultural informants provided valuable background information for analyzing 
written information about the culture. The researcher's direct experience with the 
culture also contributed to selection of translators, planning the activities of the 
study, decisions about procedural changes during the study, and analysis of the 
data. 

2. Conduct a pilot study with individuals from the target culture. Since the 
validity of findings from cross-cultural studies depends on subjects' understanding 
of their tasks, it is essential to determine how procedures or materials must be 
altered to increase subjects' understanding of the activities required of them. Some 
planned activities may be eliminated or substantially changed as a result of the 
pilot study. 

In this study, the procedures for giving the Kerby Learning Modality Test were 
altered, the interview settings were restructured, the instructional sessions were 
shortened, and the video recordings of data were eliminated as a result of the pilot 
study. On the other hand, the pilot study supported the appropriateness of media 
selected for the instructional sessions, usability of the tests, criteria used for 
selecting translators, and procedures established for interviewing subjects. 

3. Determine how subjects view their participation in the study and anticipate a 
hi gh number of noncompleters. The gap between the researcher's and subjects' 
expectations for participation first became apparent in the review of the Human 
Subjects assent form. Since this was a new experience for the preliterate Hmong 
subjects, they had no cultural context for understanding their roles as participants 
in an educational study. The fact that they did not understand how their 
participation in this study differed from their participation in ordinary English as a 
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Second Language (ESL) classes became clear in their discussions with the 
translators while the Human Subjects Assent/ Consent forms were being 
completed. 

A review of the audio tape of the study indicates that the Hmong subjects were 
confused by the legalistic wording of the consent form when it was translated into 
Hmong and did not know why they needed to sign it. They asked one translator , 
"What's this thing all about? Why do we have to sign it?" The translator then had 
to re-explain the idea of an experimental study, the form, and its wording before 
the subjects understood. 

The translators explained carefully the necessity of subjects being present all four 
days of the study and offered an opportunity for subjects to drop out of the study if 
they could not be present. As a cross-check of their understanding, the translators 
consulted with individual participants before co-signing the assent form. None of 
the subjects dropped out or asked for clarification of the need for their presence. 

Sixteen of the fifty-four subjects who began the study did not complete the 
activities due to absences. Reasons for absences included transportation problems 
and preplanned family events. One of these events was the wedding of a female 
subject's daughter to a young man who arrived in Spokane with a large group of 
Hmong refugees just released from the camps in the Philippines. 

The commitment of the potential subjects to the study on the day that they were 
asked and their later failure to keep the commitment reflect the continual cross- 
cultural tension between the orientation of traditional Hmong culture to be present 
and the emphasis of American culture on the future. Despite their intentions to 
participate in the study activities as outlined, the participants had priorities in their 
lives which subsequently affected their ability to fulfill their promise to participate 
all four days. Because they did not understand the effect of their absences on the 
study, they felt no need to inform the researcher or the translators when they 
discovered that they could not be present all four days. Their behavior may also 
have been affected by their feeling of lack of control over events in their lives due 
to their experiences with war, refugee camps, and relocation. 

4. Check the reliability of translators' messages to subjects. Involvement of 
translators and cultural informants does not insure that subjects understand what 
to do. Translators who do not understand the importance of consistency and 
clarity in explanations and procedures can contribute to inaccurate outcomes on 
tests and interviews. Such was the case with interviews performed by one 
translator used in this study. Although all translators were briefed on the 
procedures of the study, one translator had to have the assent/ consent form and 
interview questions re-explained to him by the teacher-researcher when the 
potential Hmong subjects started asking questions. 

Differences in social class and clan membership can also interfere with 
communication between subjects and translators. In this study, translators were 
either from the same Hmong clan as the subjects or from a clan which was friendly 
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to their clan. Since some Hmong clans have carried to the United States the legacy 
of their hundreds of years of animosity towards each other, attention to clan 
membership of the translators was essential to insure the participation of available 
subjects (as in the Spokane situation) and to insure comfortable and accurate 
communication between translators and subjects. 

Finally, translators cannot always be relied upon to provide the researcher with an 
accurate report on subjects' understanding of their task. It is in the interest of paid 
translators to have the researcher believe the subjects understand what is going on 
and that they have made the procedures clear to the subjects. It is, therefore, 
critical to have audio or video tapes of the study checked bv another translator (as 
was done in this study) for accuracy of translation and consistency of procedure. 

5. Accommodate cultural patterns in procedures of the study. Many of the 
procedures of Western educational research are antagonistic to the social patterns 
of some of the more traditional cultures. Cibrowski (1980) and other cross-cultural 
researchers have demonstrated that rigid adherence to Western experimental 
research procedures and materials when they clearly impede the performance of 
subjects yields studies with results of questionable validity. 

In this study, the teacher-researcher combined her knowledge of Hmong culture 
and previous experiences of this refugee population, the information from her pilot 
study of ten Hmong participants in Fresno, her constant observations of the 
behavior of the subjects, and continuous feedback from the interpreters and 
cultural informants assisting with the study to make systematic adjustments in 
procedures which interfered with subjects' performances. 

One adjustment made was in the settings for the pre-instruction and post- 
instruction interviews and tests of the subjects. Good research procedures 
normally dictate a quiet, non-distracting setting for one-to-one activities such as 
these. However, once the researcher became aware that quiet, isolated settings 
were uncomfortable for the subjects> the activities were moved so that subjects 
could be near the larger group. 

Another accommodation was made in the administration of the Kerby Learning 
Modality Test. Subjects were allowed time to clarify test directions in discussion 
among themselves after the researcher and translator had given the directions. 
This was an accommodation to the collaborative learning style of the Hmong. 

Throughout the study, the researcher and translator allowed friends, classmates, 
and family members to come and go from the area where the activities of the study 
were being conducted. The distractions that such interaction might have caused 
were offset by the comfort of subjects in the research setting. 

6. Verify the subjects' familiarity with media used in the study. This study 
demonstrated that subjects' familiarity with media affect both the way they 
respond to instructional and testing situations which involve media and what they 
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understand and learn from these educational activities. Subjects focused on 
different aspects of language when they were instructed through teacher 
demonstrations than they did when instructed with bilingual video tape. Yet the 
intention of the instructor in both situations was to promote English literacy 
acquisition. 

7. Contextualize quantifiable outcomes with the ethnographic data collected 
during the study. Without some explanations for why subjects performed as they 
did on tests and interviews in a cross-cultural educational study, their scores on the 
tests and interviews in a cross-cultural educational study, their scores on the tests 
and interviews are of little value. Without ethnographic data, the researcher could 
not have determined if subjects understood their task, were uncomfortable with 
the examiner or translator, or found the materials used in the study entirely 
foreign. In order to obtain a complete picture of subjects' performance, provisions 
must be made for collecting data on the subjects' behavior through observation 
notes, photos, audio tapes, and/ or video tapes. 

Ethnographic data in this study provided information about the Hmong subjects' 
interaction with media which no test scores could have provided. The effect of 
physical setting, the comfort level for activity and background noise, the optimal 
group size, and the subjects' responses to translators were factors which affected 
the subjects' performances on tests and interviews. Data from participant 
observation field notes on these factors helped the researcher understand such 
unanticipated subject behavior as their distaste for the photos presented. 
Observation data also helped explain why the bilingual video tape and flash card- 
photo combination were effective in promoting literacy achievement and why the 
teacher demonstrations were not. Audio tapes of the activities allowed for an 
independent check on subjects' understanding of their task and the translators' 
reliability. 

Photos provided another opportunity to gain insight into the subjects' level of 
comfort in the study setting, their attentiveness, and their way of relating to each 
other during the activities of the study. Photos also complemented descriptions of 
the physical settings given in the participant-observer's notes. 



Future Directions for Research on ESL Literacy 

In order to gain more information about the role of learner choice in the 
effectiveness of instructional media, this study should be replicated with a large 
enough group of Hmong subjects to insure that there is a selection and assignment 
group for each medium used. Only then will it be clear whether learners with no 
formal schooling understand themselves as learners well enough to choose media 
that promote their acquisition of second language literacy. 

Instruments need to be developed which can accurately measure the preferred 
perceptual modes and other aspects of the learning style of preliterate adults such 
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as the Hmong subjects in this study. Instruments presently available for 
measuring learning styles of adults require that the examinees be able to read. The 
few instruments which do not require reading, such as the Kerby Learning 
Modality Test used in this study, have been normed on young children. 

Additional ethnographic research needs to be done which focuses on the behavior 
of preliterate and illiterate adults engaged in the literacy acquisition process. 
Research to date has given only partial answers (i.e., they need instruction which 
clearly links speech and writing) to queries about the best mode of instruction. 
One area worth investigating is the appropriate level of context for literacy 
instruction, how much is needed to help learners understand meaning, and how 
much context distracts them from their task of acquiring literacy. 

Implications for Research on ESL Populations 

This study illustrates why responsible educational research on culturally different 
populations must combine ethnographic data with quantifiable data obtained 
through classic experimental methods. These data must be integrated to provide a 
complete, accurate portrayal of what learners are doing. Without an 
understanding of how and why learners behave in certain ways, meaningful 
instructional accommodations cannot be made to their individual or cultural 
differences. Such accommodations are essential if learners of different cultures are 
to achieve learning outcomes that accurately reflect their cognitive abilities. 
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TABLE 1 



Mean Scores on the Preinstruction Literacy Test (WSARPESLOPT) by Choice 
(Selected/ Assigned) and Instructional Medium (Video, Photos, Live Demonstrations) 



Selected 


20 a 


VIDEO 

M 19.44 
SD 2.45 
(9) 


Photos 

(0)b 


LWKDEM0 

M 17.18 
SD 5.88 
(11) 


111; ALL 

M 18.20 
SD 4.69 
(20) 


Assigned 


12 


M 17.20 
SD 3.83 
(5) 


M 16.57 
SD 6.60 
(7) 


(0)c 


M 16.83 
SD 5.41 
(12) 


None 


5 


M 15.00 
SD 7.41 
(5) 








Total 


37 








M 17.32 
SD 5.29 
(37) 



a The maximum score on the Literacy Section of the Washington State Adult Refugee 



Project ESL Oral Placement Test is 40 points. Subjects in this study scored below 25. 
b No subjects chose this medium. 

c Subjects randomly assigned to this medium had selected it. 
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TABLE 2 

Range of Scores on WSARPESLOPT by Medium and Choice 

















GROUP 




MAX 




MIN 




Select 




01 a 


24 


Asgn 


02 


22 


Video 


9 


15 


22 


5 


11 


20 


Photos 








7 


02 


22 


Live Demo 


11 


01 


24 








No Media 








5 


02 


20 


Total 


20 


01 


22 


17 


02 


22 



a The maximum score on the Literacy Section of the Washington State Adult Refugee 
Project ESL Oral Placement Test is 40 points. Subjects in this study scored below 25. 



TABLE 3 



Postinstruction English Literacy Achievement Test Scores by Choice 

(Selected/ Assigned) to Media with the Covariate Preinstruction Literacy in English 

(Liter) 





. : . : . : . : . : . : . : . : . : . : . >: . : . : . : . : .v. : .;.>;. : .:.:.;^.;.;.;.;.;.:.v.:.;. : .:.;.;. : . : . : . : .;. 

*" S SUM.OF • : 


m 






• SOURCE dill 
VARIATION 


MEAN 
SQUARE 


I-RATIO 


Covariate 










Liter 


U745.7A 


i 


14745.74 


27.92*** 


Choice 


245.27 


i 


245.27 


.50 


Explained 


14991.01 


2 


7495.50 


14.19*** 


Residual 


15317.98 


29 


528.21 




Total 


30308.99 


31* 


977.71 





a n = 32 The five participants who did not use media were not included in this 
analysis. 

***£<-001 
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TABLE 4 



Mean Scores on Postinstruction Achievement Test by Choice (Selected/ Assigned) and 
Media used in Instructional Sessions (Video, Photos, Live Demonstrations) 



















PHOTOS 


LIVE DEMO 




CHOICE: 












Splprtpd 


ona 


M 53 38 
SD 36.75 
(9) 


(0)b 


M 36 09 

ill ♦ V ^ 

SD 27.99 
(11) 


M 43.87 
SD 32.00 


Assigned 


12 


M 30.80 
SD 27.84 
(5) 


M 52.57 
SD 30.82 
(7) 


(0)c 


M 43.50 
SD 30.42 


None 


5 


M 7.40 
SD 15.99 








Total 


37 








M 38.82 
SD 32.07 



a The number of subjects in each treatment group is given in parentheses, 
b No subjects chose this medium. 

c Subjects randomly assigned to this medium had selected it in their preinstruction 



interviews. 
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TABLE 5 



Range of Scores on the Postinstruction Achievement Test 



$ GROUP 










MIN 


MAX 


iiimin 

•x-x:-xx:x-xx-x:x-x%vX-x-x*x*:- 


MAX 


IIBil 


Select 


20 


01 


95 


Asgn 


01 


95 


Video 


9 


01 


94.5 


5 


02 


68 


Photos 








7 


01 


95 


Live Demo 


11 


01 


95 








None 










00 


36 


Total 


20 


01 


95 


12 


00 


95 



TABLE 6 

Experience of Hmong Participants Using Audio and/or Video Tape (Reported in Years) 
for Each Media Treatment Group 



iMJllAi; 




VIDEO 


•ilPHOTOsli 


LIVE DEMO 


HI ALL 


CHOICE: 












Selected 


20 


M 9.94 
SD 9.54 
(9)a 


(0)b 


M 7.34 
SD 8.69 
(11) 


M 8.51 
SD 8.94 


Assigned 


12 


M 2.30 
SD 2.17 
(5) 


M 3.67 
SD 3.16 
(7) 


(0)C 


M 3.10 
SD 2.77 


None 


5 


M 8.24 
SD 12.59 








Total 


37 








M 6.72 
SD 8.28 



a The number of subjects in each treatment group is given in parentheses, 
b No subjects chose this medium. 

c Subjects randomly assigned to this medium had selected it. 
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TABLE 7 

Range of Years of Experience Using Audio and/ or Video Tape for Subjects in Each 
Media Treatment Group 



111 grow 


n 


• MINIMUM 


MAXIMUM 


Video 


14 


.70 


25.00 


Photos 


7 


.20 


8.00 


Live Demo 


11 


.00 


15.00 


None 


5 


.00 


30.00 


Total 


37 


.00 


30.00 



TABLE 8 

Experience Using Audio and /or Video Tape for Subjects Who Selected or Were 
Assigned to Instructional Media 





Choice 



219.51 



219.51 



4.1 r 



Residual 



1601.42 



30 



53.38 



Total 



1820.93 



31 a 



58.74 



a n = 32 The five participants who were not offered instruction through media were 
not included in this analysis. 

* p<.05 
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TABLE 9 



Mean Scores for the Auditory, Visual, and Kinesthetic Subsections of the Kerby 
Learning Modality Test (KLMT) by Selection or Assignment to Media 





■ 








AUDIT 


VISUAL 


KINES 


SELECT 


M 7.12 a 
SD 1.23 


M 9.77b 
SD 2.64 


M 7.62 c 20 
SD 1.88 


ASSIGNED 


M 6.68 
SD .99 


M 10.58 
SD 3.53 


M 7.31 12 
SD 1.52 


TOTAL 


M 6.96 
SD 1.15 


M 10.08 
SD 2.97 


M 7.51 32 
SD 1.73 


VIDEO 


M 7.16 
SD 1.00 


M 10.62 
SD 1.50 


M 7.07 14 
SD 2.06 


PHOTOS 


M 6.57 
SD .87 


M 9.94 
SD 4.50 


M 7.38 7 
SD 1.15 


LIVE DEMO 


M 6.94 
SD 1.48 


M 9.47 
SD 3.39 


M 8.13 11 
SD 1.50 


TOTAL 


M 6.96 
SD 1.15 


M 10.08 
SD 2.97 


M 7.51 32 
SD 1.73 



a Maximum possible M = 10 for the 4 part Auditory section 
b Maximum possible M = 15 for the 4 part Visual section 
c Maximum possible M = 10 for the 2 part Kinesthetic section (Auditory Motor 
Coordination and Visual Motor Coordination) 
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TABLE 10 



Range of Subjects' Scores of Subsections of the KLMT 













: 




VISUAL 


KINES 


f • \ 






HI 


Max 


Mitt 


Max 


Select 
Assigned 


4.00 
5.50 


8.70* 
8.50 


0.00 
0.00 


12.70b 
13.00 


3.00 
4.50 


10.00c 
10.00 


Total 


4.00 


8.70 


0.00 


13.00 


3.00 


10.00 


Video 
Photos 
Live Demo 


5.50 
5.70 
4.00 


8.70 
8.20 
8.70 


8.00 
0.00 
0.00 


13.00 
13.00 
12.20 


3.00 
5.00 
5.00 


10.00 
8.50 
10.00 


Total 


4.00 


8.70 


0.00 


13.00 


3.00 


10.00 



a Maximum possible M = 10 for the 4 part Auditory section 
b Maximum possible M = 15 for the 4 part Visual section 
c Maximum possible M = 10 for the 2 part Kinesthetic section (Auditory Motor 
Coordination and Visual Motor Coordination) 
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TABLE 11 



Postinstruction Achievement Tests in English (Achi) by Medium used in learning 
Sessions (Media) with the Covariate Preinstruction literacy in English (Liter) 



SOURCE OF VARIATION 


SUM OF 
SQUARES 


■ 

m 


: MEAN . 

illlilllll 


F-RATIO 








Literacy 


15161.17 


i 


15161.17 


28.04** 


Media 


4575.86 


3 


1525.29 


2.82* 


Explained 


19737.03 


4 


4934.26 


9.12** 


Residual 


17304.08 


32 


540.752 




Total 


37041.11 


36 


1028.92 





* £ < .05 

** £<.01 



TABLE 12 



Satisfaction with the Helpfulness of Media for Acquiring Literacy by Selection or 
Assignment to Media 















SUM OF 
: SQUARES 


DF 


SQUARE 


F-RATIO 


Between Groups 


24.25 


1 


24.25 


6.44* 


Within Groups 


109.10 


29 






Total 


133.35 


30 a 







a n_=31. One subject in the teacher demonstration group did not complete the 



postinstruction interview. The five Non-participants were not given the interview. 
* p_<.05 
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TABLE 13 



Satisfaction with the Helpfulness of Media for Acquiring Literacy by Selection or 
Assignment to Media 



\ 










SOURCE OF VARI ATl§§§ 


SQUARES 


•-'squares;":. 


F-RATTO 


Between Groups 


37.18 


2 


18.59 


5.41** 


Within Groups 


96.17 


28 


3.43 




Total 


133.35 


30* 







a n_=31. One subject in the teacher demonstration group did not complete the 



postinstruction interview. The five Non-participants were not given the interview. 

£<.01 
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TABLE 14 



Mean Scores for Satisfaction and Achievement 













111 GROUP ill! 


SATIS || 




ACHIEVEMENT 






11 


a 


ALL 


M 6.61 
SD 2.11 


31 a 


M 38.82 
SD 32.08 


32b 


SELECT 


M 7.31 
SD 1.89 


19 a 


M 43.87 
SD 32.54 


20 


ASSIGNED 


M 5.50 
SD 2.02 


12 


M 43.50 
SD 30.42 


12 


VIDEO 


M 5.86 
SD 2.25 


14 


M 45.32 
SD 


14 


PHOTOS 


M 5.86 
SD 1.86 


7 


M 52.57 
SD 30.82 


7 


LIVE DEMO 


M 8.20 
SD 1.03 


10a 


M 36.09 
SD 28.00 


11 



a n = 31. One subject in the teacher-demonstration group did not complete the 

postinstruction interview. The five nonparticipants were not given the interview, 
b n = 32. The five nonparticipants did not take the KLMT. 
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TABLE 15 

Range of Postinstruction Interview and Achievement Test Scores for Each Media 
Treatment Group 







A. Y-tX? 


Minimum 


Maximum 


Minimum 




Select 
Assigned 


4.00 
3.00 


9.00 
9.00 


01 
01 


95 
94.5 


Total 


3.00 


9.00 


01 


95 


Video 
Photos 
Live Demo 


3.00 
3.00 
6.00 


•9.00 
9.00 
9.00 


01 
01 
01 


94.5 

95 

95 


Total 


3.00 


9.00 


01 


95 



£ = 32. The five nonparticipants did not take the KLMT. 



TABLE 16 



Correlation of Learners' Characteristics with Achievement 



: " ■ : 
LITER 


LITER 
1.00 


ACHfcr-r > 
.70*** 


AGE 

-.37* 


SEX : 
-.08 


1 

LAND TRNG 
.27 


ACHI 


.70*** 


1.00 


-.26 


.12 


.12 


AGE 


-.37 


-.26 


1.00 


-.35* 


-.05 


SEX 


-.09 


.12 


-.35* 


1.00 


-.15 


LANG TRNG 


.27 


.12 


-.05 


-.15 


1.00 



* £<.05 
** £< 01 

*** £ < .001 
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TABLE 17 



Postinstruction Interview Scores (Satisfaction) by Location 













SOURCE OF VARIATION 


SUM OF 
SQUARES 




MEAN ■ 
SQUARE 










Between Groups 


62.15 


4 


15.43 


5.67** 


Within Groups 


71.20 


26 


2.74 




Total 


133.35 


30 







** g<.01 



TABLE 18 



Mean Scores for Postinstruction Interviews and Achievement Test Scores by Location of 
Study Site 



LOCATION 


ii* r> iilll 


SATISFACTION 


ACHIEVEMENT . 


EAC 


10 


M 4.80 
SD 1.75 


M 40.00 
SD 38.53 


PRES 


6 a 


M 8.80 
SD .45 


M 54.00 
SD 25.55 


WESL 


9 


M 7.00 
SD 1.50 


M 22.66 
SD 20.56 


MEN 


2 


M 8.00 
SD 1.41 


M 68.50 
SD 17.68 


SPOK 


5 


M 6.80 
SD 2.39 


M 66.90 
SD 18.10 


TOTAL 


32a 


M 6.61 
SD 2.11 


M 43.73 
SD 31.27 



a The subjects in this site who scored lowest on the achievement test (35) did not 
appear for the postinstruction interview. 
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Appendix A-2 



Sample of Translated Documents 
(Post-instruction Interview) 
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2M 

LUS NUG TXOG TXOJ KEV KAWM DLAW LOS 
NUMBER * LUB NPE 

Lub tswv yim kawm lus los ntawm Kaerby Modality test los: 

1. Raws lis 4 lub sij hawm koj siv los sim kawm lus dhau los 
no, koj tau siv txoj kev kawm lus twg los J awm (xws lis 
video tape, duab, kawm ntxuag tus xib hwb thiab phau ntawv). 

2. Koj nyiam siv txoj kev kawm lus ( ) 
npaum 1 i cas? . 

a . Nyiam heev 

b . Me ntsis xwb 

c. Tsis pom yuav pab tau li 

3. Koj xav hais tias txoj kev kawm no ( ) 
pab kom koj kawm tau lus npaum li cas?. 

a. Pab zoo 

b. Pab tej tsam 

c. Tsis pab li 

4. Koj xav kawm lus raus li tsoj kev kawm no mus ntxiv li cas? 

a . Kawm nraim 

b. Tej tsam los tej lub sij hawm xwb 

c. Tsis kawm ntxiv li 

5. Txoj kev twg yog txoj koj nyiam coj los siv kawm dua, yog 
kojyuavtsumrauguatusxaiv?. 

6. Muaj yam dab tsi ua kom koj nyiam txoj kev kawm lus raws 
li ( )? 

7. Muaj yam dab tsi ua kom koj tsi nyiam txoj kev kawm lus 
r aws 1 i ( ) ? 

8. Koj puas xav hais tias tseem muaj Iwm txoj kev kawm lus ua 
koj yuav nyiam dua cov koj tau xaiv kawm los no? 
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2^2 



-instruction interview of subject 
Subject tt: Name : s 

Perceptual mode identified by the Kerby -modal i ty test: 



1. Which medium did you use during the last -four learning 
sessions (video tape, photographs, live teacher with real 
obj ec ts) ? 



2. How did you like using the (name of medium) to learn 
(Engl i sh . 

a, a lot 

b. some 

c , didn't help 

3 . How much does working with (med i urn) help your liter acy 
skills in (English or Japanese)? 

a . usual 1 y helps 

b. sometimes helps 

c. doesn't help 

4. How often would you like to continue working wi-.th (medium) 
in your language class? 

a = often (e.g. dai 1 y) 
b = somet imes 
c = never 

5. Which medium would you prefer to work with, if you could choose again? 

6. What did you like about working with (medium)? 

7. What didn't you like about working with the (medium)? 

8. Do you think you would have enjoyed working with a different 
medium more than with (medium selected)? 
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Sample of Observation Notes 
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390 



apologies -for the interruption. One woman has a baby with her 
throughout the hour. She hands the baby to someone else i f she needs 
her hands -free to mark a page or handle flashcards. Be-fore the lesson 
is over, several more women leave to cook -for the school potluck. 



February 11 , 1987 
Episcopal Asian Center 

Joanna, who teaches the advanced ESL class, asks me to come into her 
class this morning and explain to them what I am doing with the study. 
She thinks it will satisfy their curiousity as well as provide a good 
1 i st en i ng comprehens i on experience. The students ask some good 
questions about my activities and are happy that I am interested in how 
Hmong learn English liter acy. They are concerned abou t why some of 
their people are learning so slowly. 

Once again, we have problems with the video portion of the video 
treatment. Lilly uses flashcards and keeps pace with the audio track. 
The students don't seem to mind that this video monitor isn't working. 
They don't have video as a regular part o-f their ESL instruction, so 
th i s is a nove 1 ty . 

We begin the achievement testing today. The people who have not 
received any treatment are tested -first. They still react to the 
-flashcards as collections o-f individual letters, rather than as words. 
M*ny simply say "] don't know" when shown a card. Some select letters 
at random from the cards and name them. A -few spell out the words. 

The students who have had the video or photograph treatment respond to 
the words on the -flashcards as whole entitities. Some are clearly 
utilizing auditory memory in responding to the words rather than 
actually reading. When they recognize, -for example, the word 'Write, 0 
they say "Write your name," even i-f the card says "Write your name on 
the paper." Word boundaries and -forms o-f written expression are clearly 
not well recognized. Students are responding the gist o-f what they 
recall to be on the cai-ds rather than the -forms they actually see be-fore 
them . 

Print style seems very important to these people. Although every e-f-fort 
is made to keep the letters consistent — all cap i tal s wr i t ten in the same 
hand and o-f the same size, the smallest changes such as a slant in print 
causes consternation. This attention to detail is seen in the -fine 
embroi dery work o-f the Hmong women . ] t may be this very thing that 
works against them when they try to read. They are so -focused on 
letters that they have trouble shifting their attention to the shape o-f 
whole words, then sentences, and so -forth. 

Private testing is not possible and may not even be desirable. Students 
seem sufficiently nervous in a one-to-one testing encounter to not need 
the added stress of isolation. They seem comfortable having other 
students hang around and look over their shoulders. There seems to be 
no carry over from the casual observat ion the testing situation to 
success in the ach i evement test . Those who 1 ook on as I test their 
classmates do no better on the achievement test than those who don't see 
the test ahead of t ime . 

1 find that confidence is a factor in completing the achievement test. 
When some of the students say "I don't know", 1 say "Yes, you do know". 
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Many of them make a second, usually successful, try after this 
chal 1 enge . 

Wesl ey Un i ted 

I am becoming adept at diagnosing video problems and fixing them. I 
bring in a new video cable to connect the player and monitor and get the 
tape to play beautifully. The video treatment group continues to be 
more attentive than the live teacher demonstration group. Today a 
woman's baby is brought in from the daycare room near th? end of the 
learning session that I am conducting. The young Hmong woman simply 
says "Her baby needs changing." The mother finishes reciting the 
particular item on the flashcard, picks up her baby and leaves. The 
woman who brought in the baby is annoyed that the mother even dealyed 
long enough to finish reciting. 

We take some pictures today. Again, my automatic camera is of interest 
to several of the students. 

Mennon i te 

Optimal group size for at ten t i veness of Hmong to a literacy lesson seems 
to be two people. The two women have no trouble focusing on the lesson. 
They do not visit and socialize while the lesson is in progress as 
usually happens when I have three women or a combination of males and 
f emal es . 

The two women who have had learning sessions score reasonable well on 
the achievement test. The 54 year old gets The forty-eight year 

old gets 48X. The two students who did not receive treatment can't 
successfully complete any of the items. 

1 learn very quickly that testing the women in an isolated situation is 
a mistake. The teacher suggests that I give the final literacy 
achievement tests to individuals in the office next door while Her does 
the post-experimental interview. I take one of the women in the "no 
treatment" group there and test her, but she is very nervous. 1 take 
her back to the room and determine to test in one corner of the room 
while Her tests in another. <I am later reminded of the traumatic 
experiences the Hmong have had related to interviews in private rooms. 
They were interrogated in their own country and they had to get through 
immigration interviews to come here.) 

Her has very good rapport with the women in the post-experimental 
interviews. He learns that two women are fairly satisfied with the 
photo treatment 1 have given, even though they did not choose it. They 
have no preference for media of instruction in the future. It is 
important to note that neither audio or video media is used in their ESL 
class. Everything is live teacher demonstr at i on , 

We take pictures with the students' permission. They look very pleased 
to be photographed. They also tell Her that they are sorry we won't be 
returning. (1 am not sure the teacher shares their feel ings. She has 
let us know nonverbally that we were an intrusion on her little class.) 

First Presby t er i an 

Lilly informs me earlier in the day that she will not be available for 
the last day of the study. She brings Lang, a middle aged Hmong man, to 
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meet me and to learn about the procedures 1 am using. He will be my 
translator all day tomorrow. 

I continue with the live teacher demon st rat i on -for the students in 
treatment groups. The no treatment group students show up at the 
beginning of the hour and have to be returned to class. An 
Engl ish-speaking aide offers to help me wi th forms and tests for a 
couple of students who have been absent for one day of this study. 

February 12, 1987 
Episcopal Asian Center 

With Lang's help, I complete the literacy tests and post-experimental 
interviews with this group. The testing has to be dnp.c in the reception 
area because the daycare area is being used by a volunteer who is 
working wi th a group from one of the cl asses. Friends and other 
students are continually hanging around and observing, 

Testing the woman with twin boys is particularly difficult. In the 
midst of the test, the boys come right to their mother's side and start 
fighting with each other. Having observed the way Hmong mothers handle 
these situations, I decide that it will be more upsetting to her if I 
stop the test than if I just continue the activity. I choose to 
continue the test. It isn't easy for either of us! Lang calmly 
conducts his interviews in the midst of this chaos. I take some final 
pc i tures and record some of the post-experimental interviews and 
achievement tests. We finally say our thank you's and goodbyes. 

Wesley Un i ted 

It is pouring rain this morning. Attendance at Weslely United reflects 
the difficulty and unpleasantness involved in gettinga round on public 
transportation in this weather. 1 lose some subjects from the study on 
this last day. Most, interestingly, are from the no treatment group. 

Lia, the bilingual aide who has helped us the last two days, provides 
invaluable assistance in getting through all the activities we must 
complete today. First he manages the video group for the last learning 
trial while I conduct the live teacher demonstration. Then he assists 
Lang in conducting the post experimental interviews and the Hmong 
translation portion of the achievement test. Again, the H no treatment 
group behaves quite differently towards words than those who have had 
weither the video sessions or the live teacher demonstration. 

Neither the selection nor the assignment group show a high degree of 
satisfaction with the learning sessions they have had. Scores are '2 
or "3"out of 10. The highest score on the achievement test is "48", 
Most score below "30 B . On a couple of test papers Vang and Lia write 
"She can spell, but she can't read. B 

First Presbyterian 

1 conduct the final learning session here before Lang and I conduct the 
achievement tests and post-experimental interviews. A large number of 
people are absent. The water at some bus stops is ankle deep by this 
point in the day. Fresno's sewer system doesn't seem able to handle 
such heavy rain. All of the subjects in my no treatment group are 
absent. Two from my treatment groups are also absent. 
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The English speaking volunteer who helped me do some testing yesterday 
casually mentions that one woman who is having trouble with the 
achievement test has dyslexia. She is surprised that the teacher didn't 
tell me. I had observed that something seemed wrong with this woman, 
but didn't know what. 

Student at this site score 30-50'/ on the achievement test. They have 
low satisfaction with the media they used. 

I take some pictures of Lang and thank him -for his help. He is 
exhausted <as am I ) « We've worked very hard since 8:30 a.m. 

The data collection in Fresno is now complete. 




Appendix A-4 



Sample Drawing by Hmong Subject 
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Figure 1 



A Mien Woman / s Drawing 
Note . Drawing produced by an ESL student at Tacoma 



Community House, Tacoma, WA 
accu 1 turat i on : A framework 
ESL classroom" by K. L. Hoi 
master's thesis, School for 



From 11 Refugees and 
for teaching culture in the 
oway , 1990 . Unpubl ished 
International Trai n i ng, 



Brattleboro, VT. Reprinted by permission 
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